We scrutinize and evaluate the rating methodologies of the big three credit rating agencies (CRAs):  S&P, Moody’s, and Fitch. We examine the factors that drive sovereign ratings, using a common regression framework, principal component analysis, and machine learning techniques with a panel of 162 countries covering ratings from 2000–2018. While all three CRAs employ complex rating methodologies based on qualitative and quantitative inputs, only a handful of variables can account for a significant proportion of the rating variation. Across all models, institutional quality is the most significant factor driving sovereign ratings, suggesting that building more vital institutions can lower a sovereign’s borrowing costs by improving sovereign ratings. Additionally, only sustainable GDP growth propelled by strong structural reforms and productive investment increase CRA ratings. We also analyze CRA rating performance and show that CRA rating changes, especially during crisis periods, are poor predictors of sovereign defaults, particularly for CRAs that rely on more subjective information (e.g., Moody’s). Finally, using machine learning techniques, we show that while the parsimonious factors in the baseline analysis have good explanatory power when retro-fitted to past defaults, they are poor predictors of future defaults. Our findings suggest that the over-reliance of market participants on CRA ratings to assess sovereign creditworthiness may be unwarranted, particularly during crisis periods.